Finding Bugs with Specification-Based Testing is Easy!

Automated specification-based testing has a long history with several notable tools having emerged. For example, QuickCheck for Haskell focuses on against user-provided properties. Others, such as JMLUnit, use specifications in the form of pre- and post-conditions to drive testing. An interesting (and under-explored) question is how effective this approach at finding bugs practice. In general, one would assume automated less bug than static verification. But, much effective? To shed light question, we consider programs written Whiley -- language first-class support specifications. Whilst originally designed verification mind, have anecdotally found surprisingly useful cost-effective. when an error detected testing, counterexample always provided. This motivated more rigorous empirical examination presented paper. that end, provide technical discussion implementation behind tool Whiley. Here, key usability concern ability parameterise input space, present novel approaches references lambdas. We then report large experiments investigating tool's effectiveness using range benchmarks, including suite 1800+ mutants. The results indicate many cases, sampling offers performance benefits only modest reductions bug-finding capability. Finally, some real-world uses where it proved (such standard library).